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ABSTRACT 

Prepared as part of a review of outcomes assessment 
programs at the six universities aid three community colleges of the 
Montana University System, this report reviews existing or planned 
assessment initiatives and provide* recommendations. Following an 
introduction, the historical context of outcomes assessment is 
described, highlighting developments at the national and local levels 
and related national legislation. Next, the current status of 
outcomes assessment in the System is discussed, focusing on the 
following elements: (1) the role ;?.nd respons ibi 1 i t i es of each unit of 

the System; (2) existing activiti-s at individual campuses, including 
proficiency assessment for students pursuing education degrees, 
student-tracking systems, and early academic intervention; (3) 
recently developed activities, including the development at one 
campus of an Assessment Committee; (4) planned activities, such as 
analyses of graduate placement and satisfaction and an economic 
impact study; and (5) current and anticipated fiscal needs. A review 
of specific issues associated with the implementation and 
consequences of assessment is then presented. Finally, specific 
recommendations for encouraging assessment activities on campuses are 
provided, emphasizing the need tc develop and implement specific 
student and teaching assessment activities to achieve institutional 
improvements and allocating fiscal resources for practices designed 
to improve learning. Contains 19 references. (TGl) 
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A Report on Outcomes Assessment in 
the Montana University System and Community Colleges 



INTRODUCTION' 

Earlier this year, staff of the Office of the Legislative Fiscal 
Analyst provided this Committee with a historic overview of the events 
which led to the creation of House Bill 0142, which provides for a 
review of the Montana University System's outcome assessment programs 
and the fiscal incentives which may be associated with the same. In 
concert with the report today provided by the Office of the Legislative 
Fiscal Analyst on the types and anticipated returns of varied fiscal 
incentives, the purpose of this Report on Outcomes Assessment i s t0 
provide the Joint Committee on Postsecondary Education Policy and 
Budget with an initial review of outcome assessment activities which 
have been implemented in recent years, or which are planned for 
implementation over the next two biennia. This Report thus provides a 
systemwide overview of activities implemented or planned at the six 
units of the Montana University System and the three community 
colleges, as opposed to a campus -by-campus examination, the later of 
which is scheduled for review by this Committee in January, 1993. Also 
included are the general costs associated with current activities, and 
the estimated costs associated with the planned activities. 

This Report is structured under four major headings: (1) historic 

context, which provides an overview of the outcomes assessment movement 
nationally and within the State of Montana; (2) a report of the status 
of current and planned assessment activities, and their associated 
costs; (3) a review of the specific issues associated with the 
implementation and consequence of assessment activities; and, (4) 
specific recommendations for encouraging the perpetuation and 
anticipated outcomes of assessment programs systemwide. 



HISTORIC CONTEST 



Overview of Outcomes Assessment . 

Assessment is a measure of performance. Changes in performance 
are measured over time; in most cases, the frequency of measuring is 
initiated (i.e., input conditions) before the object (e.g., student) 
being assessed is exposed to selected variables or conditions (i.e., 
environment/ treatment) which are believed to cause change (e.g., 
education), and again after the object has been exposed to the same 
(i.e., output conditions). Historically, assessment in an educational 
setting focused upon the academic progress and success of an individual 
student in mastering the content presented in a given course; the 
result of the assessment, determined by quantitative and/or qualitative 
means, was a recorded grade. In recent years, the context of 

assessment has been expanded to include measures of performance for (1) 
the teacher as well as the learner, (2) the value of the knowledge and 
related assignments offered to the student inside as well as outside 




I 



the classroom, and (3) the relevancy of the instructional 
experience — as it pertains to the acquisition of specific skills, 
knowledge, and values--to the demands for critical thinking, life-long 
learning, personal growth, and vocational/professional success outside 
the educational setting. 

Definition and examples. Assessment on American campuses has 
taken a number of turns in recent years, from state-mandated 
methodologies consisting of the one-time administration of standardized 
tests to enrolled sophomores or juniors, to the’ individually-designed 
and administered qualitative methodologies of portfolio development and 
capstone seminars or internships for seniors. 

Almost all public institutions of higher education in the U.S, 
collect and report data which indicate the academic preparedness of 
their entering freshmen students (i.e., admission standards which 
specify given scores on a standardized test such as the ACT); almost 
all report the admission/ graduation ratios of their student population 
over time; and, almost all report data which indicate alumni and 
employer satisfaction with the academic preparedness acquired by those 
who successfully utilized the time they invested in the educational 
experience. None, however, has provided significant evidence that the 
mere collection of such data/ alone, promotes measurable improvement in 
student, faculty, and institutional performances (AAUP in The Montana 
Professor , 1991), A few, however, have demonstrated that the process 

of acquiring and interpreting such data# in concert with having the 
time and resources to experiment with the means to enhance the 
teaching- 1 e arning relationship, can lead to enhanced opportunities for 
improved student performance as a consequence of improved institutional 
quality (Seymour and Chaffee/ 1992). 



National and local pressures to implement, monitor, and report 
assessment activities in higher education. Although the practice of 
assessment has been a part of the learning process early in the 
development of American higher education, recent calls for educational 
reform and fiscal accountability — stemming from mounting public 
dissatisfaction with the preparedness of today's college graduate — have 
increased pressures upon state legislatures, pos tsecondary educators, 
and associated parties to reexamine the teaching-learning relationship 
in America’s postsecondary institutions. By the late 1980's, some 
legislatures had mandated the implementation of standardized testing, 
the outcome of which was disfranchisement of the faculty in the 
assessment process and a lack of follow-through from the point of 
measuring to the point of fixing the inadequacies of higher education. 
Consequently, legislatures in recent years have been reluctant to 
identify the type of measurements to be utilized in the assessment 
process; however, they have been instrumental in legislatively or 
fiscally inspiring institutions to become more active in the selection 
and/or design and implementation of quantitative and qualitative 
measures of efficient fiscal management, and effective learning. In 
addition to legislative mandates, the federal government and 
independent accrediting agencies have played a significant role in 
pressuring institutions of higher education to assess themselves, and 
to utilize the findings of those assessments to improve the quality of 
the academic programs and educational experiences they offer. 
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In recent years, the Council on Po s t secondary Accreditation (COPA) 
has been instrumental in exploring the issue of educational 
effectiveness at the postsecondary level. In response, its associated 
regional accrediting bodies have individually adopted more stringent 
standards which include the development and implementation of on-going 
and comprehensive assessment activities as an integral part of the 
accreditation process; as recently as December of 1991, the Commission 
on Colleges of the Northwest Association of Schools and Colleges 
adopted new standards for the assessment of all academic programs, and 
the post-tenure review of faculty performance. Such standards will 
apply immediately to those institutions in Montana which are scheduled 
for interim or full review by the Commission on Colleges this spring 
and thereafter. 

In addition, two professional organizations which provide policy 
assistance and related input to the governing boards of state systems 
of higher education have issued reports and' provided models for 
assessing and reporting institutional effectiveness. Last year the 
Office for State Higher Education Executive Officers ( SHEEO) issued a' 
report entitled, “Assessing and Reporting Student Progress: Response 

to 'New Accountability'", and the American Association of Schools, 
Colleges, and Universities (AASCU) issued a series of reports on models 
used by community colleges for assessing institutional effectiveness. 

On November 8, 1990, the Student Right-to-Know and Campus Security 

Act was passed into law (P.L. 101-542), and requires postsecondary 

institutions to make available by July 1, 1993, and annually 

thereafter, the completion or graduation rates of certificate or 
degree-seeking, full-time students. This information must be made 
available to all students prior to their enrollment in an institution 
and/or prior to their entering into financial obligation for the 
purposes of financing their postsecondary education. 

Additional legislation, entitled "The Ability to Benefit" 
prescribes standardized testing for all students entering higher 
education without a high school diploma and who seek federal financial 
assistance. Consequently, the federal government has now set specific 
minimal standards of performance for college admission. 



CURRKHT STATUS 



In July of 1990, Executive Associate Commissioner David L. Toppen 
provided the Board of Regents with a report on Outcomes Evaluation in 
Higher Education . Although programs and their associated activities 
have been implemented on individual campuses in Montana for a number of 
years, the Office of the Commissioner of Higher Education has 
historically played an observational role in noting and further 
encouraging the development, implementation, and review of such 
activities on a voluntary basis. In light of mounting evidence that 
mandated activities and uniformly-applied methodologies fail to produce 
real correlations between the efficient use of resources and a 
measurable degree of improvements in teaching and learning (AAUP, 
1991), it is the intent of the Commissioner's Office to play a more 
highly visible and supportive role in coordinating the identity and 



development of assessment programs which are appropriate and meaningful 
to the unique missions of individual campuses, and to provide staff 
assistance in the periodic review of the results of such activities 
systemwide. As a consequence of this newly-defined role, the 
Commissioner has instituted an Intercampus Committee on Outcomes 
Assessment to oversee and coordinate the implementation, review, and 
report of outcomes assessment activities on each campus of the six 
units of the Montana University System and the three community 
colleges. 

Independent of the intercampus committee, each of the vocational- 
technical centers has developed a similar methodology of gathering and 
reporting assessment data. Currently, the Associate Commissioner for 
Vocational Technical Education, and the Executive Associate and 
Assistant Commissioners for Academic Affairs of the Montana University 
System, are independently collecting and jointly reporting the status 
of assessment activities in their responsibility areas. As a means to 
better coordinating these efforts, an invitation to extend the scope of 
the Intercampus Committee on Outcomes Assessment to include the 
vocational-technical centers is under consideration. 

Pressures for assessment have also been expressed at the state 
level. Specifically, in 1990, the Educational Commission for the 
Nineties and Beyond recommended the development of a "long-range 
program to focus the educational system on outcomes — the knowledge and 
abilities we wish our students to possess" ( Cross roads , p. 6). The 
Montana Board of Education precluded this recommendation in March of 
1989 when it developed the Montana School Accreditation Standards and 
Procedures , which offered a model of educational goals and measures 
that local districts could elect to adopt or modify to meet their 
individual needs. 



Commissioner of Higher Education’s Charge to Units of the Montana University 
System and Community Colleges . 

It is with this same respect for local autonomy and individual 
differences that the Office of the Commissioner of Higher Education has 
restrained from promoting a singular or mandated approach to outcomes 
assessment in its postsecondary institutions. Rather, the Commissioner 
has encouraged the institutions to (1) develop comprehensive assessment 
plans and activities congruent with their individual missions, (2) 
utilize the data gathered in the assessment process to identify 
weaknesses and strengths of their institutional performances, (3) 
identify and implement policies and processes which encourage improved 
performance of a quality approximate to their peer institutions, and 
(4) provide documentation that such improvements have been realized 
under appropriate and self-imposed time lines. 

Role of the Intercampus Committee on Outcomes Assessment. For the 
purpose of improving the effectiveness and efficiency of public higher 
education in Montana, the Intercampus Committee on Outcomes Assessment 
(ICOA) shall be comprised of one representative from each of the six 
units of the Montana University System and the three community 
colleges, and shall be charged with the responsibility of providing 
assistance collectively to the four-year and community college campuses 
in the development of selected and comprehensive assessment programs 



appropriate to the unique missions of the various institutions, by (1) 
acting as a clearinghouse for the acquisition and distribution of 
literature on new developments in assessment, the discussion of 
alternatives for implementing specific assessment activities, and the 
communication of the perceived effectiveness of various assessment 
programs; (2) encouraging the coordinated development of such programs 
through the setting of reasonable time lines and the establishment of 
jointly agreed upon protocols of reporting; . and, (3) periodically 
reviewing the campuses' assessment programs and activities, and 
offering suggestions for improvement. 

The Committee's initial list of commonly-recognized issues include 
the availability of valid and reliable data for individual and 
comparative assessments of institutional practices influencing (1) 
retention, (2) accessibility, (3) the appropriate match between the 
student and institution, (4) the appropriate match between two-year and 
four-year institutions, as regards the efficiency and effectiveness of 
transfer curricula and related policies/procedures, and (5) the 
appropriate match between graduates from two-year and four-year 
institutions and the workplace/society. 

In the development of a systemwide approach to these issues, the 
ICOA will solicit and interpret information which pertains to current 
or anticipated activities in each of the commonly-recognized areas, 
providing proposed schedules for the timely examination of each, as 
well as the anticipated fiscal impact of initiating, implementing, 
continuing, and reporting activities associated with full assessment 
and improvements in each area. In addition, individual representatives 
will periodically report to the ICOA the status of any current or 
proposed special programs and related activities outside the five 
commonly-recognized areas. 



Strategies and time lines for reporting system-wide status of 
assessment implementation and results. At its September 1991 
organizational meeting, the Joint Committee on Pos tsecondary Education 
Policy and Budget adopted the following schedule for the hearing of 
reports germane to the status and progress of outcomes assessment in 
higher education: 

February, 1992: Commissioner of Higher Education's 

presentation on Montana Project Status — 
initial status report of activities systemwide 

May, 1992: Report and review comments on Montana’s 

assessment activities, presented by the staff 
of the Legislative Fiscal Analyst 



January, 1993: Commissioner of Higher Education's progress 

report on activities implemented and planned 
campus -by- camp us 



Status Report of Current and Planned Assessment Activities Svstem-wide . 

Each institution has initiated a task force for identifying and 
implementing specific assessment activities, and for establishing time 
lines for the reporting of outcomes. In addition, each institution has 
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a person or persons assigned to the responsibility of institutional 
research; consequently, the ability to gather and interpret data for 
assessment purposes exists systemwide. However,, the ability or 
willingness of institutional researchers to inform institutional 
leadership of the significance of such findings and to otherwise 
influence decisions made by institutional leadership with regard to the 
link between assessment findings and improvements in educational 
effectiveness remains unknown, and possibly unappreciated. 

Furthermore, a systemwide methodology for soliciting and compiling 
progress reports on the status of implementing assessment activities 
and the success in improving educational effectiveness is a new 
responsibility of the Intercampus Committee on Outcomes Assessment 
(ICOA). 

Therefore, for the purpose of providing the Joint Committee with 
an initial and general progress report on the outcomes assessment 
activities of Montana’s four-year and community college campuses, this 
Report will provide (1) examples of long-standing assessment activities 
on five campuses, (2) examples of recently developed and soon-to-be 
implemented activities on one campus, and (3) examples of planned 
activities on the remaining campuses. 

Long-standing assessment activities. For one campus, a series of 
outcomes assessment procedures have been implemented over the years for 
measuring the academic performance of undergraduates pursuing degrees 
in elementary or secondary education. Specific assessment of student 
proficiency in the major includes (1) demonstration of competencies in 
specific courses (e.g., psychology, English, anthropology) with a grade 
of C or better, (2) completion of the National Teachers’ Examination 
with a minimum score of 648 in communications skills, and a minimum 
score of 644 in general knowledge, (3) the well-written, and 
professionally-oriented essay, and (4) satisfactory completion of a 
formal interview with three faculty members. Following these 
assessments, students receive a written evaluation of their performance 
during the junior year field experience. Following successful 
achievement towards degree completion and professional acceptance, 
students apply for admission to student teaching, which provides 
standards for the completion of courses in the professional sequence 
with grades of C or better, complemented by a grade point average of 
2.5 or better in the major and minor fields, and a cumulative grade 
point average of 2.2 or better. Following their student teaching 
experience, students receive a performance report. In addition to 
meeting institutional requirements for graduation, teacher education 
majors must also complete the professional knowledge section of the 
National Teachers' Examination at or above the cutoff score for 
certification in the State of Montana. Graduates of the program are 
interviewed the year following the completion of their degree to 
determine their satisfaction with the quality of their academic 
program; findings are communicated to the teacher education faculty for 
the purpose of identifying strengths and weaknesses of the program 
which may be altered to improve educational effectiveness. 

Another campus secured sponsored research funds in previous years 
to develop a competency-based curriculum in a specific graduate 
program. Other programs responded to pressures from their individual 
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accrediting bodies to implement specific self-assessment activities. 
In 1984/ the campus established a formal , campus-wide student-tracking 
system for eventually informing decisions with regard to specific and 
general education reviews. By 1989/ the campus established a task 
force of deans and department chairs, and named a faculty member to an 
administrative post for the purpose of gathering information on the 
latest developments in student assessment and to formulate a plan for 
developing a comprehensive student assessment program on campus. 
Attention will be given to four primary areas of concern: (1) goal 

achievement, (2) improving institutional quality and excellence, (3) 
creative experimentation in the development of instruments and the 
application of assessment findings, and (4) the coordinated autonomy 
for faculty in planning and revising the curriculum for the purpose of 
improving effectiveness. In addition, the task force has recommended 
several important elements: (1) the appointment of a special assistant 

to the academic vice president and provost for continuity in the 
coordination and implementation of an comprehensive assessment program, 
(2) the establishment of a central repository of technical information 
made available to faculty involved in designing specific assessment 
instruments and activities, and (3) focus upon three areas of students' 
academic and social lives — (a) general education, (b) education in 
majors and programs, and the (c) quality of student life. Trial 
projects will include development of an assessment program for an 
Honors College, with expectation that assessment activities will extend 
eventually to the departments which supply instruction to the Honors 
program. 

In addition to the evaluation of individual student achievement in 
general education and major studies, a third campus currently engages 
in assessment activities for math placement, improved advising 
procedures, retention, and campus diversity. Other activities include 
the assessment of student performance by external examiners for the 
certification of educators (i.e.. National Teachers' Exam) and 
accountants (i.e.. Certified Public Accountants' Exam); furthermore, 
business faculty are involved in gathering baseline data on the 
academic performances of entering and graduating students. 

The fourth campus has engaged in assessment activities which 
mirror the five commonly-recognized areas of concern to the ICOA. 
Specifically, the campus has initiated intervention processes for 
identifying students who are experiencing academic difficulties by 
mid-term of their first year, and offering tutorial assistance. With 
regard to the issue of accessibility, courses which are offered less 
than once every two years are evaluated for possible elimination from 
the curricula. Special student orientation is designed to help 
entering and transfer students acclimatize to the campus environment, 
and faculty are encouraged to evaluate the relevancy of their 
instructional programs to the needs of graduates in the workplace and 
society. In addition, the faculty are encouraged to actively 
participate in the campus' overall strategic planning activities as 
they address curricular reform and educational effectiveness. 

The fifth campus, which has utilized College Board's standardized 
lacement tests for entering students over a period of ten years, has 
eveloped a baseline for determining the levels of reading 
comprehension, writing strengths, and math proficiency of its new 
students. High risk students receive remedial intervention, and some 
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thought has been given to the implementation of competency-based exit 
tests for associate degree graduates. 

Recently developed and soon-to-be Implemented activities. One 
campus , in preparation for a review of its assessment activities as a 
condition of continued accreditation, established an Assessment 
Committee last year. The Committee identified three primary goals of 
its planned activities: (1) better retention, (2) better program 

planning, and (3) better interfaces between feeder schools and the 
receiving institution. The Committee met with the campus president to 
identify resources needed to implement a successful assessment program. 

Planned activities. Three campuses, following formation of the 
ICOA, have formed campus task forces for the purpose of overseeing the 
development and implementation of assessment activities. One intends 
to establish a student/tracking demographic reporting system by the 
close of this academic year; a second intends to analyze existing 
academic programs as regards placement and alumni satisfaction, and to 
implement an economic impact study to demonstrate the financial effect 
a postsecondary institution has on its surrounding communities. A 
third campus intends to divide its assessment activities into two 
initiatives: (1) student development and outcomes, and (2) 

institutional effectiveness. A fourth campus intends to initiate an 
inventory of current campus assessment activities in the Fall of 1992, 
to review campus options for the development of assessment activities 
in the Winter of 1993, and to develop intended student outcomes and 
objectives at the departmental level by the close of Summer of 1993 . 



Current and Anticipated Fiscal Needs System-wide . 

Early assessment activities have been funded by sponsored programs 
such as FIPSE and Title III. One institution subtracted one-percent of 
the allocations made to each department and pooled the resources in a 
fund for intercampus grants, one of which supported a selected 
assessment activity. Another institution has estimated that costs for 
the development and implementation of a comprehensive assessment 
program will cost the equivalent of $100 or more per enrolled student. 

Although proponents of fiscal incentives may argue about the worth 
of rewarding institutions which demonstrate effective learning, the 
necessity for developmental funding has been expressed by all members 
of the ICOA. Furthermore, the Education Commission for the Nineties 
and Beyond recommended that the Board of Regents 

develop a budget for the Montana Assessment Project for 
presentation’ to the 1991 Legislature and the Legislature should 
support this project with a special appropriation that recognizes 
its innovative nature and importance. These funds should be used 
principally for instruction, released time and summer compensation 
for faculty involved with this project ... we estimate this 
effort will require a decade from start to completion .... 

( Cro s s roads , p. 6). 
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SPECIFIC ISSUES 



Historically, the phrase outcomes assessment has been associated 
with student performance, focusing upon with the difference which 
occurs in students cognitive and social development between the time 
they enter and the time they exit an educational institution. 
Traditionally, assessment which took place initially in kindergarten 
through high school settings, and later in higher education was 
reported in the form of academic preparation (i.e*, success in meeting 
admission standards), grades assigned in individual courses, the 
composite academic requirements necessary for graduation, and the 
relevancy of the preparation students received for entering the 
world-of -work or further education; such application and reporting was 
based on the assumptions of a scientific model which held that once the 
student entered the system, s/he continued to receive educational input 
(i.e., treatment) until s/he exited (i.e., graduated) as a changed 
(i.e., educated) product. Today, such assumptions are no longer valid 
because students in kindergarten through high school do not necessarily 
graduate from the system which initially accepted and educated them, 
and students in higher education do not necessarily start their college 
experiences immediately after high school graduation, nor do they 
necessarily progress without interruption between the time they enroll 
in and graduate from one or more institutions; consequently, the 
development of specific skills, the acquisition and application of a 
breadth of knowledge, and the maturation to a preferred set of values 
may occur inside as well as outside instructional and non-instructional 
settings, both on and off postsecondary campuses. Thus, a multitude of 
dependent and independent variables must be recognized when determining 
the effect classroom experiences play exclusively in shaping the 
outcome of the learning-teaching relationship measured by student 
outcomes assessment. 

Student performance measured by the classroom activities and 
assignments, however, neither describes fully nor accurately the 
multi-dimensional and interrelated opportunities for ’’learning" within 
the college experience — i.e., is it truly possible to separate the 
acquisition of knowledge about values, which may be presented in class, 
from the encul turation and internalization of those same values in the 
socialization process which takes place in campus life outside the 
classroom? 

In addition, it is also necessary to examine and appreciate the 
relationship between effective teaching and effective learning. Thus, 
student performance must be assessed in terms of its relationship to 
faculty and institutional performance. The implementation of 
assessment for the acquisition of data alone will not result in 
improved quality; thus resources which are only aimed at the 
development and implementation of assessment activities will be 
unwisely allocated. Additional steps are needed: resources must be 

made available to interpret the data, to make policy and procedural 
changes which enhance the performance of students academically, and 
which ensure the ongoing improvement of faculty and institutional 
performance . 
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Time, too, will play a factor: change in academic performance is 

a function of growth, and growth is measured over time. If 
institutions are expected to be candid in measuring their effectiveness 
in positively influencing academic performance and students’ capacities 
for life-long learning, they too must be given the opportunity (without 
fiscal penalties) to change and measure their growth over time. Thus, 
abbreviated windows of time for the assessment and the implementation 
of measures to ensure change must be avoided whenever and wherever 
possible. 



SPECIFIC RE COMMENT) AT I OHS 

FOR ENCOURAGING ASSESSMENT ACTIVITIES ON CAMPUSES 



The data collected as a part of the assessment process should be 
interpreted in an environment of legitimate concern for changes in the 
policies and practices which impact efficient and effective learning. 
The collection of data for the sole purpose of accountability will not 
result in improved educational effectiveness; it will result only in 
improved efficiency in the collection of requested data, which may or 
may not be of value in determining and affecting educational 
effectiveness . 

Thus, in principle, the development and implementation of specific 
student and teaching assessment activities must be directed to the 
purpose of achieving institutional improvements while promoting and 
protecting the involvement of faculty and students in all dimensions of 
assessment; in turn, the assessment of overall institutional 
effectiveness must involve not only the faculty and students of the 
individual campuses, but also the administrative and governing bodies 
of the campuses and System. 

Finally, the fiscal incentives associated with the identification, 
implementation, and outcomes of such studies should not be used to 
penalize those institutions which find a need for improvement; rather, 
such fiscal allocations should be utilized for the adoption of 
practices which improve learning. ' Comprehensive efforts to improve the 
quality of instruction, the relevancy of the learning experiences 
to students, and the preservation of those services and 
missions unique to the individual campuses of the Montana Systems of 
Higher Education will be dependent upon and shaped by the timely 
availability of adequate resources. 
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